not sure if page or chunk is better, but it should be clear this is a limit for the chunk and not a total limit for the async iterator

This is actually the total limit for the whole iterator. Chunk size is limited by the length of the platform response; it is not limited by this code.

so we are yielding the whole pages? i thought we would yield just the items, one by one

Well, my idea was that the iterator should yield exactly the same type as the original response, so that you can use the same code to process it. We also already have the whole chunk in memory. For example:

actors = await apifyClient.store().list({ limit, offset });
processActors(actors)

for await (const actorsChunk of actors) {
    processActors(actorsChunk)
}

But I guess you would prefer?

actors = await apifyClient.store().list({ limit, offset });
processActors(actors)

for await (const singleActor of actors) {
    processSingleActor(singleActor)
}

If we return the full pages, it will make the usage harder (forcing users to use nested loops). I would yield items one by one, not pages.

Updated to allow iteration over individual items, while making API requests only for as big chunks as possible.
So for example when using limit 3000, it will make 3 requests (3x1000), but for await loop will run 3000x times.

I have a feeling this could be simplified using the pagination fields from the response.

How about e.g.

const isLastPage = Math.min(page.total, options.limit + options.offset) <= page.count + page.offset

This works because:

$n = \text{page.count} + \text{page.offset}$ tells that the current page includes the $n$-th list item. if $n = \text{page.total}$, we've seen the entire list.

The options.limit and options.offset options select an interval of the list between

$$(\text{options.offset}, \text{options.offset} + \text{options.limit}\rangle$$

If the current page includes the

$$n = \text{options.offset} + \text{options.limit}$$

$n$-th item, it's the last page containing any items from the selected interval.

Then we could IMO do

let page = await fetchPage({ limit: options.limit, offset: options.offset });
yield* page.items;

/// The variable names are rather for explanation, not production ready
const lastItemIndex = Math.min(page.total, options.limit + options.offset);
let lastItemIndexFromThePreviousPage = page.count + page.offset;

while (lastItemIndexFromThePreviousPage < lastItemIndex) {
    const remainingItemCount = lastItemIndex - lastPageItemIndex;
    page = await fetchPage({ limit: remainingItemCount, offset: lastItemIndexFromThePreviousPage });
    lastItemIndexFromThePreviousPage = page.count + page.offset;

    yield* page.items;
}

Having typed this out, I'm no longer convinced this is a better way... but feel free to get inspiration 😅

Regarding pagination fields from the response. According to the documentation, they are not defined on all list endpoints. I think we can rely only on total and items. (Previously, I was under the impression that even total was optional, but re-checking the documentation, it seems it is always there.)

Here is an example of the minimal endpoint: https://docs.apify.com/api/v2/act-versions-get
(Tested and it does really return just items and total)

Therefore, I would define the algorithm in a way that does not require any optional fields from the response. But knowing I can get total allows at least some of the improvements you suggested.

I would also keep this in while condition currentPage.items.length > 0 condition, to handle any API problems or situations when the requested resources change during the iteration due to some external action. (Someone removing an actor could otherwise lead to an infinite loop)

const lastItemIndex = Math.min(page.total, options.limit + options.offset);
Since those fields are optional, this would not work for a user who defines just an offset.

https://api.apify.com/v2/acts/.../versions is not a paginated endpoint though; it just lists all the items under items and the length of this array under total (see implementation here). See that it won't react to offset nor limit query params.

All paginated endpoints will use the Pagination class, which means those will always accept all the parameters (and return all the fields).

I made it in a way that can be applied to all the classes that were internally calling ResourceCollectionClient._list, even the "fake paginated endpoints".

I'm thinking if now all the XYZOptions interfaces could extend this interface to reduce the duplicate code

see e.g.

Some of these interfaces are wrong. For example:

export interface ActorEnvVarCollectionListOptions {
    limit?: number;
    offset?: number;
    desc?: boolean;
}

But the endpoint does not take any parameters (tested, and also the documentation does not mention parameters)

https://docs.apify.com/api/v2/act-version-env-vars-get

I applied it to all the options and raised an issue with the platform people about the unused options
https://apify.slack.com/archives/C01VBUV81UZ/p1764238228809389

btw.
StoreCollectionListOptions does not have desc as the only option there :-(

Maybe we could have a generic utility type, sth like

type PaginatedListIterable<T> = Promise<PaginatedList<T>> & AsyncIterable<T>

to reduce the amount of code and simplify adding new endpoints?

Now I saw IterablePaginatedList<Data> from utils.ts, why not use that?

That would work for the normal ones, but there are these weird ones that I am not sure how to fit into such new type.

Normal one:
Promise<PaginatedList<DatasetCollectionClientListResult>> & AsyncIterable<DatasetCollectionClientListResult> == PaginatedListIterable<DatasetCollectionClientListResult>

Weird ones:

• Promise<Pick<PaginatedList<ActorEnvironmentVariable>, 'total' | 'items'>> & AsyncIterable<ActorEnvironmentVariable>
• Promise<Pick<PaginatedList<FinalActorVersion>, 'total' | 'items'>> & AsyncIterable<FinalActorVersion>
• Promise<PaginatedList<Omit<KeyValueStore, 'stats'> & { username?: string }>> & AsyncIterable<KeyValueStore>
• Promise<PaginatedList<RequestQueue & { username?: string }> & { unnamed: boolean;}> & AsyncIterable<RequestQueue>

I applied it to the normal ones, but the special ones will stay special.

I have no strong opinion on the name.

(not that it would matter much in here)

Done

Let's merge this thread with #790 (comment)